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Observable Changes in Skin and Mucous Membrane 



BACKGROUND OF THE INVENTION. 
1 . Field of the Invention. 

10 

The invention, in general, relates to a method and to an apparatus for 
measuring and classifying, in predetermined categories, optically observable 
changes in the skin and mucous membrane. More particularly, the invention 
relates to the measuring and classifying of such changes by endoscopic 
15 observation. 

A preferred field of application of the invention is the classification of skin 
anomalies such as, for examples, melanomata and melanotic changes in respect 
of benign, malignant or suspicious characteristics or properties. Other fields of 

2 0 application are the classification of anomalies of skins and mucous membranes 
of endoscopically observable cavitary organs, such as the oesophagus, the 
stomach, the duodenum, the colon, the rectum, the bronchial system, the 
cavities of the throat, nose and ears as well as the bladder. Such anomalies may 
be, for instance, diverticula, adenoids, carbuncles, tumor infiltrations and 

2 5 tumor compressions. 



Malignant melanomata and the like may at the present time be identified 
with certainty in their late stage, when they are extremely dangerous and usually 
fatal, in their early stage, they may, however, be successfully treated in a 
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relatively simple manner. At this point in time it is, however, very difficult to 
identify or diagnose them. That is to say, it is difficult at this time to classify skin 
anomalies as malignant or benign melanoma. It is the task, therefore, of the 
treating physician to classify skin anomalies into a group of harmless skin 
5 anomalies (benign melanomata and other benign non-melanocytic skin changes) 
- and a group of unambiguously malignant or subjectively suspicious melanomata^ 
It is possible, of curse, to observe or diagnose several skin anomalies for any 
one person. To ensure that not a single melanoma is overlooked the treating 
physician must, at the slightest suspicion of malignancy, classify the patient as 
10 one to be treated further. 

This kind of approach will of necessity lead to a very large group of 
patients who are at risk. This, in turn, results in the motivation to improve the 
diagnosis by the use of commonly available means to allow qualitatively 
15 excellent classifications of skin anomalies, especially of melanomata, even by a 
physician who does not have the requisite expertise to classify merely on the 
basis of visual data. 

2. The State of the Prior Art. 

20 

For the registration of skin anomalies and melanomata in particular it has 
been known to irradiate a two-dimensional test area of the skin by light in a first 
range of wavelengths suited to generate fluorescence and in the wavelength 
range of the fluorescent light to generate an image of the test area (see German 
2 5 patent DE 4,026,821 Al). However, an evaluation of the fluorescent image by 
itself is not sufficiently reliable for classification in view of the fact that the 
topology of the skin surface in the test area exerts a strong influence on the 
intensity distribution of the fluorescent image. As disclosed by DE 4,026,821 Al, 
the influence of the topology of the skin surface is to be reduced. To this end the 
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test area is irradiated by light of a second range of wavelengths, and a reference 
image is formed with light of the second wavelength range. By then scaling the 
values of intensity of the fluorescent image to those of the reference image, an 
imageimay be obtained in which the changes in intensity caused by the topology 
5 are substantially eliminated and which is suitable for diagnosis by the treating 
. ... physician^. ^ . ; , . ......... .~. ~ — 

Also, a fluorescent diagnosis device suitable for testing tumors in the 
internal wall of body cavities is known from German patent DE 198 19516 Al. 

1 0 Basically, the device consists of an endoscope and a camera for providing 
images of the test area on a monitor as well as of a spectrometer for 
spectrometricaily examining the fluorescent light generated by selected sites of the 
test area. This diagnostic device does not, however, process the image to 
simplify the task of the diagnosing physician. Furthermore, the topology of the 

15 tested tissue area has no effect upon the diagnosis. 

W09747235 discloses a diagnostic system and method suitable for 
quantifying the visual appearance of skin anomalies, such as melanomata, by 
digitally analyzing images of the skin. In a preferred embodiment, the system 

2 0 includes a high-resolution digital camera, a computer, a data storage as well as a 
color monitor. The camera generates an image of the skin area to be examined, 
as well as, close to the area of the skin to be examined, of a longitudinal scale 
and of a color scale. This makes it possible to scale color values and spacings 
in the image independently of illumination and exposure conditions. The 

2 5 computer defines and subsequently displays on the monitor, the external contour 
of dark skin areas. Also, characteristic values of the color of the examined skin 
area as well as two-dimensional geometric characteristic values such as size and 
asymmetry of the external contour of dark skin areas may be derived, and 
characteristic values and images generated at different times may be compared, 
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by the diagnostic system. Even though the diagnostic system of W09747235 
provides different characteristic values of geometry and color, an identification of 
skin anomalies is nevertheless strongly dependent upon the experience of the 
examining physician. 

5 

For rendering the classification of skin anomalies more objective and for t 
automating it, it is known for purposes of classification to process difference 
characteristics of skin anomalies such as shape and color in color images of a skin 
area by a neuronal net (Ercal, F.: et al., IEEE Transactions on Biomedical 

10 Engineering, (1994 Sep.) 41(9) 837-45). For classification, the method strongly 
relies upon the processing of characteristic values of color as well as upon the 
asymmetry and irregularities of the marginal contour of a skin anomaly. The 
method does not make use of the size or the diameter of a skin anomaly, or of 
their changes over time, as characteristic values during the classification 

15 process, and it requires a very precise definition of the margins of skin anomalies 
which for that reason is performed by a dermatologist. 

Another example of the use of a neuronal net for classifying tumors in 
ultrasound images has been described in U.S. Patent 5,260,871. In this case, 
2 0 characteristic values are extracted from the ultrasound images and input into the 
neuronal net which performs a classification. This solution, too, does not evaluate 
data relating to the three-dimensional structure of a suspicious area of 
tissue. 

25 An essential disadvantage of known solutions resides in the fact that no 

data about the three-dimensional structure of suspicious tissue areas is being 
evaluated and used for purposes of classification, It has, however, been shown that 
for evaluating melanoma examinations, three-dimensional structure and 
growth in height are important criteria for determining the property of 
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melanomata. 



OBJECTS OF THE INVENTION. 

5 Accordingly, it is an important object of the invention to provide a method 

. and an apparatus suitable for measuring optically observable changes of skin . 
and mucous membrane and which include the three-dimensional structure of 
such changes and their changes over time as important classification 
characteristics. 

10 

It is another object of the invention to measure and classify skin 
anomalies such as melanomata and melanoma-like symptoms by taking into 
consideration their three-dimensional and temporal changes. 

1 5 BRIEF SUMMARY OF THE INVENTION. 

As used herein, the term "skin" is intended to connote not only the 
external skin but also mucous membranes of a person or animal. 

20 In accordance with a currently preferred embodiment the invention 

provides for a method of measuring and classifying optically observable changes 
in skin by, firstly, taking and recording at least one image of a change in the skin 
suitable for defining the three-dimensional structure and color structure and at 
least one reference image of at least the vicinity of the change, by, secondly, 

2 5 computing the surface measurements of the changes in skin consisting of the 
three-dimensional coordinates of the changes and of their associated color 
values scaled to the reference image and of, optionally, computing the changes 
over time, by, thirdly, performing a classification, preferably by means of a 
neuronal net, on the basis of the computed surface measurements together, 
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optionally, with other data relevant to a patient such as age, type of skin, 
allergies, dermatoscopic or sonographic diagnoses, by, thirdly, performing 
another classification, particularly in a case which defies certain classification, on 
the basis of the previously obtained data but differing from the previous 
5 classification by the use of a modified algorithm and/or by using a selection of 

- . . .,. comparative data,. preferably evaluated by experts,, stored in at least one data 

base, and by, fourthly, issuing the results of the classification yielded by the third 
and fourth operations. 

1 0 The apparatus in accordance with the invention preferably includes a 

measuring head or cameras for generating at least two photogrammetrically 
evaluatable images of changes in skin and a pattern projector for projecting onto 
the area of the skin to be examined a pattern suitable for detecting three- 
dimensional data of the area, at least one of the cameras being suitable for 

1 5 recording as reference images for defining a reference color images of an area 
of healthy skin in the vicinity of the changed skin, a first computer for preparing, 
processing and storing the images generated by the measuring head and, more 
particularly, for computing surface data of a change consisting of the three- 
dimensional coordinates of the change and the associated color values scales to 

2 0 the reference image, and for computing changes over time of these data, a 

second computer for executing a first classification of any changes on the basis 
of the surface measurement data and, optionally, of the stored relevant data 
relating to a patient such as age, type of skin, allergies, dermatoscopic or 
sonographic diagnoses, etc., optionally, a third computer for performing, with a 

2 5 modified algorithm and/or a selection from a data base of comparative data 

evaluated by experts, a further classification, particularly in the case of defying 
* certain classification, on the basis of the surface measurement data and, 
optionally, of the stored relevant data relating to the patient. 
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The method of the invention is based upon the concept of generating in 
intervals of times appropriate for a diagnosis and by means of suitable recording 
techniques, such digital images of changes of the skins to be examined and 
optically observable which can be evaluated in respect of the three-dimensional 
5 structure of the depicted subject as well as of its color structure. A recording 
. ...technique, suitable for this purpose is known from the.present inventors', German. ~ 
patent 196 23 172. 

In a first operational step, the surface measurement data of changes in 
10 the skin and mucous membrane are computed from the digital images. The data 
consists of the three-dimensional coordinates and the associated color values 
scaled to the reference image. Optionally, the changes over time of the changes 
are additionally computed. 

15 In a second operational step, the surface measurement data are 

evaluated, preferably by a neuronal net, by a process of classification. The 
result will be a classification into, for instance, benign, suspicious or malignant 
changes. 

20 If it is not possible to obtain a certain classification on the basis of the 

image material provided in accordance with the invention and the surface 
measurement data derived therefrom, a third operational step will be performed 
in accordance with another basic concept of the invention. The third operational 
step comprises a classifying process differing from the previous one by a 

2 5 modified algorithm and/or by the use of a selection of comparative data, 
preferably evaluated by experts, stored in at least one data base. 

In this data base, evaluated histories of diseases are documented by 
images recorded at different dates and by three-dimensional data as well as by 
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data relevant to the disease. It is of utmost importance that the records of these 
histories of diseases have been classified with a very high degree of certainty, 
frequently by histological examinations. Instead of the classification of the 
changes in skin and mucous membrane, or in addition to the classification, an 
artificial neuronal net connected to the data base may can select comparative 
material from the data base, i.e., histories of diseases of changes in. skins and.,, 
mucous membranes the physical properties and appropriate relevant data of 
which are comparable to the changes in the skin and or mucous membrane to be 
examined. From the selected comparative data suitable images including 
evaluations which are similar to the input data may be selected and returned to 
the treating physician. 

Execution of the operational steps is possible in the following variations: 

♦ all operational steps are being performed on a local computer; 

♦ the first and the second operational step are performed on a local 
computer and the third operational step is performed by a central 
computer; 

♦ the first operational step is performed on a local computer and the second 
and third operational steps are performed on a central computer; 

♦ all of the operational steps are performed on a central computer. 

In accordance with a further advantageous embodiment of the inventive 
method at least one of the data bases containing comparative data evaluated by 
experts is disposed and maintained on the central computer, the local computer 
having access to the central computer. 

The apparatus in accordance with the invention comprises two major 
components: A measuring head for generating at least to photogrammetrically 
evaluatable images of the changes in the skin to be examined and classified and 
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at least one computer connected to the measuring head by way of data lines or 
integrated into the measuring head. 

The measuring head is provided with at least two calibrated cameras, 
hereinafter sometimes referred to as camera group, for generating digital images 
„of the area to beexamined, and with a pattern projector for. projecting onto, the,- ^ 
area to be examined at least one pattern suitable for detecting three-dimensional 
data of the said area. The camera group is adjusted such a surface of 
appropriate size may be measured by it. At least one of the cameras of the 
camera group is adapted for taking color images, this camera or, optionally, a 
further camera, being additionally provided for taking, as a reference image for 
defining a reference color, an image of an inconspicuous area in the vicinity of 
the change in the skin. The measuring head is movable and may be positioned 
at an appropriate location. 

The at least one computer connected to the measuring head by way of 
data lines is provided with means for preparing, processing and storing the digital 
images of the skin and mucous membrane changes generated by the measuring 
head. Such means may include means for computing data of the three- and 
two-dimensional structures of the changes and their color structures relative to 
the reference image as well as of the temporal changes of the said data. The 
computer is additionally provided with means for carrying out a classification on 
the basis of the previously mentioned detected data and, if necessary, the 
relevant data relating to a patient such as age, type of skin, allergies, 
dermatoscopic or sonographic diagnoses etc. A neuronal net may be used to 
provide such classification. 



For use in connection with a case which cannot be classified with 
certainty, the computer is preferably provided with means for carrying out a 
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further classification with a modified algorithm and/or a selection of comparative 
data evaluated by experts, on the basis of the previously detected data and, if 
necessary, the relevant patient-related data stored in a data base. 

5 In an advantageous embodiment of the device in accordance with the 

invention there are, provided, at least two computers. interconnected by. data. lines. 

At least one of the computers is a local one and at least one other computer is a 
central one. In such a case only the central computer is provided with the means 
applied in a case which cannot be classified with certainty such as, in particular, 
1 0 the data base of comparative data evaluated by experts. 

DESCRIPTION OF THE DRAWING. 

The novel features which are considered to be characteristic of the 
15 invention are set forth with particularity in the appended claims. The invention 
itself, however, in respect of its structure, construction and lay-out as well as 
manufacturing techniques, together with other objects and advantages thereof, 
will be best understood from the following description of preferred embodiments 
when read in connection with the appended single drawing which is a flow 
2 0 diagram schematically depicting the cooperation between the operational steps 
in accordance with the invention. 

Other objects and features of the present invention will become apparent 
from the following detailed description considered in connection with the 
25 accompanying drawings. It is to be understood, however, that the drawings are 
designed as an illustration only and not as a definition of the limits of the 
inventions. ~ ~ 
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The example shown relates to the classification of skin anomalies known 
as melanoma and melonocytic changes into suitable categories such as, for 
instance, benign, suspicious or malignant. 

5 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS. 

The measuring head_2 comprises a camera group 5_of two calibrated color 
cameras for generating digital images and a pattern projector^. Where 
necessary, the pattern projector^ makes it possible to illuminate skin anomalies 

10 by unstructured light, i.e. at a substantially constant light intensity, with one light 
pattern or with a plurality of sequential light patterns. The lenses and the 
positions of the camera group_5 and of the pattern projector_4 are chosen so that a 
surface measuring about 4 x 4 cm may be measured by them. A surface, e.g., a 
ringjS, with reference colors constitu e s constitutes a further component of the 

15 measuring head_2. This is an accessory for efficiently comparing results of 
measurements provided by different measuring heads. 

To obtain the images necessary for determining the mcasurment 
measurement data of skin anomalies the following procedure is deemed 
20 appropriate: 

Initially a color image is recorded of a healthy area of skin in the vicinity of 
a skin anomaly. This is a reference image and serves to define the actual basic 
skin color (basic complexion) of a patient. The color data of a skin anomaly may 
25 be scaled relative to the reference image. 

Thereafter, the measuring head_2 is positioned such that the skin anomaly 
7 is preferably positioned in the central measuring range of the camera groupji. In 
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the case of two or more closely adjacent skin anomalies it may be useful to 
deviate from the rule such that these skin anomalies are possibly detected by a 
single position of the measuring head_2. A color image, hereinafter sometimes 
referred to as a light image, in which the skin anomaly is illuminated with 
unstructured light may then be taken by at least one camera of the camera 
-.group.5. This image serves to define the actual color values of the skin.change. _ 

Furthermore, without changing the position of the measuring head_2, 
images are recorded in which one or, sequentially, several appropriate light 
patterns are projected onto the skin anomaly by the pattern projector^. Such light 
patterns are indispensable to the process. 

The recorded digital images or image sequences form the basis of the 
computation of the data relating to the three- and two-dimensional structures of 
the skin anomaly and its color structure relative to the reference image. To this 
end, a local computer 1L proceeding from the recorded digital images 12, 
computes in a first processj or operational step surface measurement data of the 
skin anomaly. The surface measurement data consist of the three-dimensional 
coordinates and associated scaled color values of the surface of the skin 
anomaly. 

The images or image sequences of the skin anomaly taken with the light 
patterns are used to define the three-dimensional coordinates. Known or novel 
algorithms used in image processing are used for the computation. The 
algorithms are based on the definition of corresponding pairs of pixels by 
matching processes or by correlation algorithms. One possibility of calculating 
surface points is based on the principle of block matching. Small image segments 
having image contents that agree with one another are pushed on top of one 
another with as much of the same coverage as possible. 
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If an "image sequence of the skin anomaly" is recorded, the "correlation 
algorithm" described below can be used; 

The sequence of gray values of an individual pixel that are der ived from 
the images taken one after the other in a time sequence serves as the basis for the 
calculation..The principle, of the measurement consists of finding two pixels ,. 
having gray value progressions that agree with one another as much as possible. 
The cross-correlation coefficient between the two gray value progressions can be 
used as a criterion of agreement. The pixels having the maximal cross-correlation 
coefficient are determined by way of an iterative search process. 

The light image and the reference image are used to define the scaled 
color values of the skin anomaly. The color values of the light image are affected 
by the activity of the skin anomaly as well as by the basic complexion of the skin 
(depending on a seasonally conditioned degree of tanning). To minimize the 
effect of the last-mentioned affect scaling relative to the reference image is 
useful. Scaling may be carried out in the conventionally (use of an offset and 
lightness correction) or by the neuronal net manner. A possible "conventional 
method" consists in first comparing the color values red, green, and blue of the 
border regions of the "light image" and of the "reference image" with the values 
of the "Ring of Reference Colors" 6. It is presupposed that these skin regions are 
healthy. For this purpose, the difference as compared with the values of the "Ring 
of Reference Colors" 6 is formed for the "light image" and for the "reference 
image." The offset is obtained by subtracting the difference value of the 
"reference image" from the difference values of the "light image." 

Utilization of an artificial neuronal network can take place in the 
following manner: a two-layer or three-layer network is trained with different 
examples of "different seasonally conditioned degree of tanning." One then 
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obtains the correct color values during the measurement at the output. The result 
will be scaled color values. Correlating the scaled color values to the three- 
dimensional coordinates is simple since both data are defined on the basis of 
images from the same cameras in the same positions. 

5 

^The surface measurement data 13 of the skin anomaly determined jn this 

manner are subjected in a second operational stepjl also performed on a local 
computer to a first classification process 15 by means of a neuronal net 14. The 
neuronal network can be presented in simplified manner as a multi-layer system. 

10 At the beginning, pixels are present. In a first layer, characteristics data 

appropriately assigned to the image in two dimensions are made available by 
means of signal generation devices such as filters, for example. These 
characteristics data are linked in a neuronal network, by means of linked 
neurons, in a second layer, to produce combinations of characteristics that are 

15 still assigned to layers in two dimensions, in accordance with the image 

structure. The individual combinations are summed up within the layers (in 
other words over the image area), thereby resulting in a characteristics vector in 
a third layer, which vector contains different combinations of characteristics 
summed up over the image area. This characteristics vector is compared with a 

2 0 reference vector, for the purpose of classification . This results in a correlation 
of appropriate categories, e.g., benign 15a , malignant 15b or suspicious 15c . 
This result and the surface measurement data obtained are well suited for 
documentation and archiving in a patient data base_8. If measurements are taken 
at different times, the relevant changes of the surface measurement data of the 

2 5 skin anomaly can be determined from the documentation. 

In order further to improve the quality of classification a telematic system 
16 may optionally be utilized. In such a system the surface measurement data of 
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the skin anomaly measured by the treating physician J_ are transferred to a central 
computer 17 and correlated to or compared with relevant patient related data_9 
obtained elsewhere, such as age, type of skin, allergies, dermatoscopic or 
sonographic diagnoses and so forth. 

. r ... A third operationaLor process step IILmav then.be carried out on the . 
central computer 17. The third step consists of a further classificatio n 18a similar 
to the second operational step II based on modified algorithms and/or a selection 
of suitable comparative data from at least one data base. 

The data base may be a central data base 20 stored in a central computer 
17 . Alternatively or additionally, the data base may be an external data base 19 
which may for instance be accessible through the Internet. 

Such data banks contain evaluated histories of diseases documented by 
images and surface measurement data recorded at different times as well as 
relevant patient-related data_9 (described supra) relevant to the disease. It is 
decisive that these disease histories have been classified with a high degree of 
certainty, often by histological examinations. 

On the basis of data stored in the mentioned data bases the artificial 
neuronal net 18 or another suitable algorithm of the central computer 17 will 
detect skin anomalies the surface measurement data of which are sufficiently 
similar to the skin anomaly measured by the treating physician J, and transferred to 
the central computer 1 7. The patient-related data are also considered. In this 
manner, it is possible to obtain suitable comparative data 23 such as evaluated 
' images 22, 23a and optional additional dat a 2 K 23b such as histological 
diagnoses and other data of interest from the history of disease. They are returned 
to the treating physician! • They serve to improve the result of the classification. 
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Another telematic application resides in consulting an expert 10a to whom 
the surface measurement data, the result of the classification and relevant patient- 
related data are submitted by data lines. Such consultation may be carried out 
in the manner of video conferencin g 10b . 

.......... Further possibilities of .use of the inventive method and. apparatus, arise in 

connection with the following problems: 

♦ The measuring head may be utilized for augmenting diagnoses of other 
diseases in the field of dermatology. A typical application resides in the 
measuring and evaluation of skin rashes. Another application is the 
measuring changes of color and height caused by allergy tests. Precise 
documentation is made possible by the measuring head. 

♦ For documenting and evaluating the healing progress of wounds 
measurements may be very useful. To this end, two-dimensional and 
three-dimensional data as well as the color structure are measured at one 
or more points in time. Analogously, measurements may be taken of skin 
burns where a precise definition of the degree of burn is of importance. 

♦ Radiation treatments may also result in changes in the skin color of a 
patient. Where such changes are measured it is possible to arrive at 
conclusions as to the radiation sensitivity of the patient. 

♦ The measuring head may also be used for measuring and evaluating skin 
creases or wrinkles (primarily in a face). This may be important to the 
cosmetics industry in connection with determining, for instance, the 
effectiveness of skin creams and lotions. 
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